Email – The Tech Channel That Testing Forgot but We Can Save

Email marketing has always been a tech-forward discipline. We email marketers are not pen-and-paper marketers—we’re digital natives. From the early days of personalisation through dynamic content to the Generative AI explosion, we have embraced technology like few other channels.

We have email rendering platforms, drag-and-drop editors and deliverability tools to create stunning emails that render correctly across inbox clients and show up reliably in our subscribers’ inbox. And lest we forget, email pioneered AI-driven personalisation long before it was cool.

So why—why—didn’t we evolve our A/B testing tools? And what can we do to catch up before it’s too late?

The glaring gap: How A/B testing tools fell behind

In a world that built a multi-million dollar industry around Conversion Rate Optimisation (CRO), I’m baffled that email testing is still so rudimentary.

Most ESPs offer A/B testing tools. But to be brutally honest, they fall short in these six key ways:

• Overly simplistic. They test only subject lines or CTAs, not the effectiveness of one entire message against another.
• Statistically unreliable. They use flawed winner-selection logic like 10-10-80. (Sending the control email to 10 percent of the audience, the variable to another 10 percent, and the supposed winner to the remaining 80 percent within a short non-statistically significant period of time.)
• No educational support. And I mean none, zero, nada, forget about it, to help marketers design strategic, hypothesis-driven tests.
• No multivariate testing: A/B testing isn’t just control-versus-variable. An A/B/n structure is valid if your hypothesis justifies it, but the tools don’t let you try.
• Dead-end results: The whole point of testing is learn what works and what doesn’t. But these tools don’t help you tie your results to these deeper strategic insights.
• No ongoing education or easy recording documentation: You have to build your own tracking framework in Excel or Google Sheets to track results and discern trends.  That means no automatic pattern detection, no long-term learnings, and a whole lot of manual grunt work for insights that should be effortless.

CRO has left email in the dust

Look at what’s happening in web and landing page optimisation. Whole teams, agencies, and platforms are dedicated to testing everything: layout, messaging, user experience (UX) flow, button color and placement, testimonial positioning.

Here’s the CRO team:

• Specialist CRO consultants
• Dedicated experimentation platforms like VWO, Optimizely, and Convert.com
• Standard processes for hypothesis creation, sample sizing, and statistical validity

Meanwhile, over in email? Many email marketers don’t have access even to valid 50-50 audience splits, let alone the tools to analyse why one version worked better than another.

Why did email drop the ball?

It’s not that email marketers don’t care about testing. In fact, a LinkedIn poll I ran showed that 44% of respondents struggled with creating a strong hypothesis and 33% struggled with sample size calculation. This goes to show that the desire is there, but the tools and education aren’t up to the job, and the vendors have been slow to improve them. Why?

A few hypotheses:

• Legacy systems within ESPs impede modernisation
• Low user demand: if people don’t know what’s possible, they won’t ask for more
• Short-term thinking: testing is seen as a campaign tactic, not a long-term strategy
• Fragmentation: no unified solution has stepped in to fill the gap… yet

And then we have the overarching issue: the chicken-and-egg problem. The tools we have don’t let marketers test properly. (See my list of grievances above). This often renders the results inconclusive or misleading. Over time, marketers become disillusioned, stop testing, or treat it as a box-ticking exercise instead of a powerful learning tool.

AI shows us what we’re missing

Imagine if our testing tools could:

• Handle multi-variable hypotheses
• Guide you through creating a valid test, with automated sample sizing and significance
• Analyse results across multiple campaigns to detect patterns
• Explain not just what won, but why—based on audience behaviour, psychology, and historical trends

This is exactly what we achieve with our Holistic Testing Methodology, enhanced by GenAI. We’ve trained our own private GPT, whom we have nicknamed “Chad,” to help marketers test strategically, reduce guesswork, and extract insights that matter.

AI can connect the dots between tests, simulate outcomes, and suggest what to test next. This turns testing from a tactical bolt-on into a strategic growth lever.

But here’s the thing: Our agency has advanced faster than other marketers, both in how we use AI and our experience in A/B/n testing. We’ve spent more than a year training Chad and building our system. Most marketers don’t have the time, technical expertise, or infrastructure to create this kind of workaround. They need tools that are ready to go, not just the raw AI potential. That’s where the gap widens even more.

Final thought: It’s time to demand more from email testing

As marketers in the email channel, we should be ashamed of how far behind we’ve fallen in testing innovation. We’ve embraced AI, personalisation, automation, even schema. But testing—the one thing that ensures we’re making smart decisions—is still clunky, manual, and shallow.

We deserve better.

Marketers must demand tools that match email’s technological sophistication. Let’s stop accepting the status quo. The next time we sit in a pitch meeting with a platform asking us for our business, let’s demand the testing tools we need.

Better yet – let’s not wait for someone to solve our problem for us. Let’s build our own. We can call on teams of smart engineers and developers to design tools that elevate email to the strategic powerhouse it truly is.

Are you with me?